Make doc store binary format more compact (#903) - #3009
Conversation
Store field ids and integers in the doc store using variable-length encoding instead of fixed-width, reducing block size before compression: - Field ids: fixed 4-byte u32 -> VInt - u64: fixed 8 bytes -> VInt - i64: fixed 8 bytes -> zig-zag + VInt (new zig_zag_encode/decode helpers in tantivy-common) - Dates left as fixed 8-byte nanos (VInt would grow modern timestamps) Introduces doc store format V3. The change is backwards compatible: V1/V2 segments still deserialize via the original fixed-width path, selected by the version stored in the footer. On a numeric-heavy schema (20k docs x 100 u64 fields + 1 i64), the LZ4-compressed store shrank from 12,714,578 to 9,445,506 bytes (~25.7%). Adds a zig-zag roundtrip unit test and a pathological-schema store roundtrip test.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 034c4c5d13
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
|
||
| /// Doc store version in footer to handle format changes. | ||
| pub(crate) const DOC_STORE_VERSION: DocStoreVersion = DocStoreVersion::V2; | ||
| pub(crate) const DOC_STORE_VERSION: DocStoreVersion = DocStoreVersion::V3; |
There was a problem hiding this comment.
When a pre-V3 segment is merged, the merge path can copy raw document bytes into the new store (iter_raw/store_bytes) or copy compressed blocks via stack, but this version bump makes the new merged segment footer advertise V3. Those copied V1/V2 document bytes are then decoded with the V3 field-id/integer layout, so stored fields from upgraded indexes can become unreadable after a merge even though opening the original segment still works. The merge code needs to avoid raw stacking/copying when the source StoreReader version differs from DOC_STORE_VERSION, or reserialize the documents into V3 before writing the V3 footer.
Useful? React with 👍 / 👎.
| /// Same as V2 but field ids and integers are stored using variable-length | ||
| /// encoding (`VInt`, with zig-zag for signed integers) to make blocks more | ||
| /// compact before compression. | ||
| V3 = 3, |
There was a problem hiding this comment.
Introducing V3 here changes the on-disk store bytes, but INDEX_FORMAT_VERSION is still 7 in src/lib.rs. In a rollback or older-reader scenario, previous Tantivy versions will accept the index metadata as a supported format and only fail later when stored documents are loaded because their DocStoreVersion enum cannot decode version 3; bumping the global index format makes the incompatibility fail cleanly at open time instead of leaving a partially readable index.
Useful? React with 👍 / 👎.
| // Starting with `V3`, field ids are stored as a variable-length integer | ||
| // rather than a fixed 4-byte `u32`. | ||
| let field = if self.doc_store_version >= DocStoreVersion::V3 { | ||
| let field_id = VInt::deserialize(self.reader)?.val() as u32; |
There was a problem hiding this comment.
For a V3 store whose field-id vint is corrupted or produced by a non-conforming writer with a value above u32::MAX, this cast silently truncates it to another Field instead of reporting data corruption. That can make stored values appear under the wrong field when the wrapped id happens to be valid, so this should validate the vint with a checked u32 conversion before constructing the Field.
Useful? React with 👍 / 👎.
1 participant
Store field ids and integers in the doc store using variable-length encoding instead of fixed-width, reducing block size before compression:
Introduces doc store format V3. The change is backwards compatible: V1/V2 segments still deserialize via the original fixed-width path, selected by the version stored in the footer.
On a numeric-heavy schema (20k docs x 100 u64 fields + 1 i64), the LZ4-compressed store shrank from 12,714,578 to 9,445,506 bytes (~25.7%).
Adds a zig-zag roundtrip unit test and a pathological-schema store roundtrip test.